In this video we learn how to convert time series from one frequency to another frequency with the free
sample method.
So when our temperature data set we have hourly data and in that time span of three years we are having
a lot of timestamps and data points and we have also seen that high frequency data looks quite messy.
And we tried to plot and visualize it and therefore I'd a normal workflow to aggregate data from higher
frequency to lower frequency.
So in our example we could convert our temperature data from our lead to a daily or monthly and converting
time serious data from high frequency to low frequency is also called down sampling.
All right so let's first of all import pandas and we are also importing met plot lip and we use of the
Seabourn style.
And then we also import our temperature data set.
And of course we set a date time index.
So let's have a look here.
The first five rows and the info method.
So give you f on the left hand side our daytime index.
And as we can see here we have our lead temperature data and we can convert the hourly data flight sample
to daily frequency.
And we can do this service at the very power for the re sampling method.
And actually the recycling method works in a quite similar way then the group by method.
So first of all we apply the re sampling method on our temperature data frame and then let's have a
look here inside the brackets.
So here we have a pyramid I call it rule.
So let's have a look here and here we have to pass the offset string or object representing target conversion.
So here we can pass the frequencies like daily weekly annual or whatever so we can use our frequencies
that we learned also in the last videos and TV on to transform or convert our time series to daily so
we pass the idea.
And if you now understand we are getting actually an object called daytime index a re sampler.
So this is quite similar to a group by up checked and but there is sample method actually DOES IT groups
or timestamps on our data frame in two respective groups.
And in this case here in two daily groups so far each single day here in our data from the Reid sample
method creates one group and the group contains all timestamps of the respective day and same s with
a group by object that can also convert here the day time index every sample object into a list by using
here the list function
and here we are getting a list with one data frame for each day in our data frame so our data frame
consists of three years of data.
So approximately the amount of days is 1400.
So actually here our lists would contain about 1400 data frames so let's check this also with the length.
Function
so here we have one thousand four hundred sixty one elements on our list.
And uh let's have a look at the one specific element here.
So for example for the very first element so here we can see the very first element in our list and
each element is actually a top layer consisting of two elements.
So the first eleven this year the information about the day.
So here we have the first of January 2000 and 13 and the second element is actually a data frame and
this data frame contains all timestamps for their respective days.
So in our example the first of January 2013.
So in total we have here in this data frame twenty four timestamps for 24 hours.
So the very last timestamp is 11:00 p.m. So this is our very first element in our list and this is a
topic that we can also select actually the data frame and the top US.
So this is the second element here at an exposition one and TBA actually the data frame for the first
of January two thousand and 13 with twenty four timestamps
and you can also slice our list for example for the second group so far the second day.
So yeah it's the 2nd of January 2013.
Also give us the 24 timestamps and actually we can do this for 1400 something days so this was actually
the first part or the first step of down sampling data.
So we are grouping our data by specific rules so for example daily.
And now actually in a second step We have to define how to aggregate the data.
So for each day we have 24 timestamps and 24 data points and we want to convert these 24 data points
into a one data point so folks and we can take the very first data point at midnight or the last data
point at 11:00 in the evening while we can take the mean or the median or whatever and we can aggregate
our data quite easily by changing an aggregation method or function to the recent low object.
So here we are creating a reasonable object and then we have to apply and that creation function or
method on it.
And here in this example we want to have for each day the very first times.
At midnight so we can do this here
and we can see now our risk armpit or down sample data frame and on the left hand side we can see that
now we have a daily frequency starting from January the 1st.
Then we have January the 2nd and so on and the temperature data here for each day is actually the very
first timestamp at midnight because we apply to the first method.
So let's again have a look at our original temperature data frame
and maybe let's have a look at the first 25 rows and we can see on January the 1st to the very first
time stamp at midnight that we have eleven point seven degrees and a minus one point one in New York
and exactly this information or this data we have here now and our reason but data frame.
So the very first time stamp on January the first year at midnight and actually here we have many options
to aggregate so we can also select here the very last timestamp of each day.
So let's have a look at the very last time stamp here we have 11:00 p.m. And here we have four 14 degrees
in a and two point seven in New York.
And here we have it and also a very common aggregation method all functions of the mean.
So you're paired us calculates that the mean over our twenty four time stamps in a day.
Oh we could also calculate the median
and in.
We could also take here the sum of all or 24 timestamps for values.
So in this case here with our temperature data this does not really make sense but in other cases this
might be helpful.
All right.
So now let's go through some other examples of some different frequencies and you can also convert our
hourly data into increments of two hours and we can take the very first time stamp of each increment
here.
So here we have the period from midnight to 2:00 and you take the values at midnight then give you have
the period from 2:00 to 4:00 give us the values at 2:00 and so on and of course we can also transform
our data to a weekly data by passing here W and we are taking get the mean value over all values
and by default we can see here on the left hand side that the last day of each respect the Feig and
the last day is a Sunday.
So we have here the week until the 6th of January it's a Sunday.
And the average temperature in L.A. was ten point two one.
And then we have the week until the 13th of January.
And the average temperature was ten point four nine.
And so on and the last day of a particular week is here by default always Sunday.
But we can also customize this and we can say that we want to have a week from first territory Wednesday
and therefore we can pass yet w Wednesdays.
So let's try this out here.
So now we can see here on the left hand side to the end of each week.
And I'll be at the end of each week as the Wednesday because now we have grouped our data to a weekly
date with Greeks from Thursday to Wednesday.
So here the second to say Wednesday and also the 9th.
So Wednesday and so on.
So let's move on to a monthly frequency and therefore we pass here hmm
and by default we can see here the month end.
So for example here the 30 first of January and the twenty eight of February and the value ECI actually
the aggregated mean values for the respective month.
So here we have the mean temperature so for January for February and so on.
And instead of having here the very last day of the month of the label you can also change this to the
very first day by passing year.
As for start
and actually the value you see in our data frame do not change.
So these are the aggregated and mean data for January and also here we have the same data.
The only thing that is different this year on the left hand side to the labels.
So here we have as the label it's the very last day.
And yes the very first day.
So first of January first of temporary and so on and we can actually further customize them the label.
And they said the perimeter offset.
So let's have a look here
and here with our offset we can actually adjust the reasonable time labels and let's assume that for
each month we want to f the 15 s uh the label here then we can positive for example 14 days.
So we are starting from the very first day off a month and we are adding 14 days and then we are ending
up at the 50s.
So let's try this out here
and still here the value is in our data frame are the same as here above.
So this does not change.
The only thing that does change actually here the label is.
So here we had the very first day of each month you we had the very last day and here we have the 15th.
All right so now let's go also to quarterly data and we can pass a Q
and only by default.
We are getting the very last day of a quarter.
So for example here the thirty first of March then the 30th of June then the last day of September and
the last day of December.
So this is the default setting and having quarters from January to March April 2 to in July to September
and October to December.
But we can also change this and we can define for example that the very last month of a quarter should
be February.
So now you'll be half the quarter and the twenty eighth of February then we have the quarter and thirty
first of May.
And so on.
So essentially we have four quarters from March to May and then from June to August and so on.
So this is we're flexible actually and we can also respond to our data frame to a yearly frequency by
passing year y or alternatively a
and by default we have fear in the index the very last day of the period.
So for example here we have the aggregated data for the year 2013 and we have here the very last day.
So the thirty first of December but we can also customize see of the labor by passing y s fourth stop
date
and here we have actually the same values again in our data frame but we simply changed t on the left
hand side of the labels from the very last day of the year to the very first day of the year.
So these were some examples how to read sample a data frame and there are even more options.
And feel free to have a look at the documentation to see the full options but uh for the time being
we are not finished.
And I hope to see you also in the next video by.